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Description 

Field of the Invention 

[0001 ] The present invention relates to a method and 5 
apparatus for allowing a consumer at point-of-sale to 
evaluate colour cosmetic products on their virtual facial 
image prior to purchase. 

BACKGROUND OF THE INVENTION AND PRIOR 
ART 

[0002] Colour cosmetics encompass among others 
lipsticks, eyeliner, blush, llpliner, eye shadow and facial 
foundation. A vast array of colours and shades are avail- 
able to consumers. Often the consumer finds it difficult 
to select among this vast array of choices. It is difficult 
to decide how a particular colour or shade will look when 
actually spread on the skin. Even more difficult to assess 
is how a combination of colour cosmetic products such 
as facial foundation, lipstick and eye shadow will work 
together for a particular individual. 
[0003] Systems have been developed to assist the 
decision process. Clinique and Clarion have installed 
computers at sales counters for use by the consumer. 
Information on colour shade, oiliness and other proper- 
ties of a consumer's skin are punched into the computer 
which then determines the company's most closely 
matching product. 

[0004] Another point of sale technique has been that 
of custom blending. Two major companies, Prescrip- 
tives (division of Estee Lauder) and Visage (division of 
Revlon) begin a sale by manually evaluating a subject's 
skin colour. The sales person then adjusts existing fin- 
ished foundations so as to match the evaluated skin col- 
our for a perfect facial foundation. Unfortunately, the 
system is time consuming, requires extensively trained 
sales persons, and has poor reproducibility. 
[0005] U.S. Patent 5,854,850, U.S. Patent 5,825,941 
and U.S. Patent 5,687,259 all to Linford et al. assigned 
to the Mirror Software Corporation disclose a system for 
digital image capture of a particular person's face and 
software to manipulate facial structures. The system Is 
intended for reconstructive plastic surgery. 
[0006] JP-A-63 0B0 692 describes a makeup simula- 
tion system whereby a face is imaged via a camera onto 
a television monitor. Colour makeup can then be select- 
ed for app I cation to different parts of the image. 
[0007] It is evident that relatively unsophisticated sys- 
tems are available for colour matching. The technology 
also exists for digital recording of individual faces. To 
date there has been no disclosure of systems, which uti- 
lise digital imagery for cosmetic products. Neither has 
there been reported any systems for obtaining virtual 
makeover of a consumer's image whereby the consum- 
er can see the effect of different colour cosmetic prod- 
ucts on their own face. 

[0008] Accordingly, it is an advantage of the present 



invention to provide a method and apparatus for forming 
a virtual makeover of a person's face on an image mon- 
itor so as to allow selection of the best colour combina- 
tions. 

[0009] Another object of the present invention is to 
provide a method and apparatus for virtual makeover at 
a point of sale counter wherein colour makeup can be 
electronically painted over the consumer's image with- 
out requiring actual removal of make-up prior to obtain- 
ing a baseline natural skin colour image. 
[0010] These and other objects of the present Inven- 
tion will become more apparent from the following sum- 
mary and detailed description. 

SUMMARY OF THE INVENTION 

[001 1] According to a first aspect, the present inven- 
tion provides a method for applying a virtual makeover 
to a person's face, the method involving: 

(i) directing a digital camera at a person's face to 
register an image of at least a portion thereof; 

(ii) calculating colour parameters on two or more ar- 
eas of the image to identify a natural colour of the 
skin; 

(iii) transmitting the image to a monitor for displaying 
the face; 

(iv) correlating the natural skin colour with a prede- 
termined palette of colours appropriate to the cal- 
culated natural skin colour; 

(v) locating areas of the face for application of a col- 
our cosmetic; and 

(vi) displaying the facial image with the predeter- 
mined colour palette on the located areas. 

[0012] According to a further aspect, the present in- 
vention provides apparatus for applying virtual makeo- 
ver to a person's face, the apparatus including: 

(i) a digital camera for acquiring information on the 
person's facial image; 

(ii) a program for calculating colour on two or more 
different areas of the facial image not normally cov- 
ered by cosmetics to identify a natural colour of the 
person's skin; 

(iii) a monitor for displaying the facial image; 

(iv) a program for painting a predetermined colour 
palette correlated with the natural skin colour onto 
certain located areas of the face normally subject 
to colour cosmetic treatment; and 

(v) a printer delivering a printout of the predeter- 
mined colour palette. 

DETAILED DESCRIPTION OF THE INVENTION 

[001 3] A problem with prior attempts to visualise col- 
our cosmetics for customers has been the inability to 
determine their natural skin colour. Now it has been 
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found that natural skin colour can be determined via dig- 
ital imaging by comparing L*a*b colour values on at least 
two different areas of the facial image. Consumers do 
not place makeup over 100% of their face. The digital 
scan identifies those areas of skin which are not covered 
by makeup, provides a colour value for that area and 
utilises the value as a baseline colour for re-imaglng a 
cosmetically stripped face onto a monitor. 
[001 4] Another challenge for the system was to locate 
those areas of the face upon which colour cosmetics can 
be applied electronically. Theprogram locates colour re- 
quiring areas such as the lips, cheeks and eyes. 
[001 5] A digital image of a customer's face may be 
taken at a point-of-sale department store counter for 
storage into a central memory and then made available 
at a website. The image may also be storable locally at 
the store or within an intranet system of the cosmetic 
supplier. Also envisioned is the alternative possibility for 
the customer scanning their own actual image from their 
home computer. Many of these customers now have rel- 
atively inexpensive digital imaging cameras interfacea- 
ble with personal computers and thereby can connect 
directly to the internet. 

[0016] Once the natural skin colour has been identi- 
fied, a consultants choice Is automatically determined 
from a software stored set of 12-20 colours matching 
the person's skin type. This provides a colour matching 
facial foundation for building the first stage of colour 
makeover. Feedback is.then accepted into the program 
on the customer's own preferences. For instance, these 
preferences can be with respect to three possible 
"looks" such as fashion, natural or go-to-work types. 
[0017] A lipstick can then be provided as a consult- 
ant's best choice. The customer can also provide input 
for her favourite colour. The pick may be a natural or 
dramatic look. These choices are then fed directly into 
the program for facial display on the monitor. 
[0018] Other colour cosmetics can be applied in the 
same consultant's choice and consumer preference 
feedback mode. These cosmetics can include eye shad- 
ow, eyeliner, lipliner and blush. The monitor can display 
images from no makeup to full makeup with ail products 
or any step in between. 

[0019] When a particular look including all the neces- 
sary colour cosmetics have been selected, a printer is 
activated with an ordering slip for the selected cosmet- 
ics. The customer is identified by a number and the se- 
lected products given a barcode symbol or other desig- 
nation correlating the product with the customer. 
[0020] The customer's image can be placed on an in- 
ternet site. At some subsequent time, the customer can 
view the Internet site and from the comfort of their own 
terminal evaluate various makeovers on their face. Cos- 
metics to accomplish the actual makeover can then be 
ordered on the web site or through any other channel. 
[0021] Equipment for the method includes a digital 
camera available from several sources. Nikon provides 
a suitable high resolution digital camera underthe trade- 



mark Cootpix 900™ providing an image size of 1 080 x 
960 pixels, VGA mode 640 x 480. Another digital cam- 
era, which may be utilised, is the PhotoGenie™ sold 
with the ePHOT0 1 680 camera by the Agfa Corporation. 

5 Image size is 1 280 x 1 200 pixels , VGA mode 640 x 480 . 
Software for operating the system can be obtained from 
Torchlight Corporation of Seattle. Programs upon which 
this software is based are found in U.S. Patent 
5,854,850, U.S. Patent 5,825,941 and U.S. Patent 

10 5,687,259. 

[0022] The foregoing description illustrates selected 
embodiments of the present invention and in light there- 
of variations and modifications would be suggested to 
one skilled in the art, all of which are within the scope 

15 of this invention as defined in appended daims. 



Claims 

20 1. A method for applying a virtual makeover to a per- 
son's face, the method comprising: 

(i) directing a digital camera at a person's face 
to register an image of at least a portion thereof; 
25 (||) calculating colour parameters on two or 

more areas of the image to identify a natural 
colour of the skin; 

(iii) transmitting the image to a monitor for dis- 
playing the face; 

30 (jv) correlating the natural skin colour with a 

predetermined palette of colours appropriate to 
the calculated natural skin colour; 
(v) locating areas of the face for application of 
a colour cosmetic; and 

35 (vi) displaying the facial image with the prede- 

termined colour palette on the located areas. 

2. A method according to claim 1 further comprising 
placing an order for colour cosmetics correspond- 

*o ing to the predetermined colour palette. 

3. A method according to claim 1 or claim 2 wherein 
the located areas are lips and in step (vi) displaying 
on the lips a lipstick mark or a colour other than of 

45 the natural colour of the skin. 

4. A method according to claim 1 or claim 2 wherein 
the located areas are around the eyes and in step 
(vi) displaying around the eyes an eyeliner or eye 

so shadow mark of a colour other than the natural col- 
our of the skin. 

5. A method according to any of claims 1 to 4 further 
comprising the step of assigning to a customer, rep- 

55 resented by the person's face, a number and as- 
signing to the colour cosmetics a barcode symbol 
or other designation correlating ordered colour cos- 
metics with the customer. 
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6. A method according to any of claims 1 to 5 further 
comprising inquiring with a customer represented 
by the person's face on their own colour preferenc- 
es which input is then registered to change the col- 
our of the located area of the displayed facial image. 

7. An apparatus for applying virtual makeover to a per- 
son's face, the apparatus comprising: 

(i) a digital camera for acquiring information on 
the person's facial image; 

(ii) a program for calculating colour on two or 
more different areas of the facial image not nor- 
mally covered by cosmetics to Identify a natural 
colour of the person's skin; 

(iii) a monitor for displaying the facial image; 

(iv) a program for painting a predetermined col- 
our palette correlated with the natural skin col- 
our onto certain located areas of the face nor- 
mally subject to colour cosmetic treatment; and 

(v) a printer delivering a printout of the prede- 
termined colour palette. 

8. An apparatus according to claim 7 wherein the lo- 
cated areas are lips and the painting program dis- 
plays on the lips a lipstick mark having a colour oth- 
er than that of the natural colour of the skin. 

9. An apparatus according to claim 7 wherein the lo- 
cated areas are around the eyes and the painting 
program displays around the eyes an eyeliner or 
eye shadow mark of a colour other than the natural 
colour of the skin. 

10. An apparatus according to any of claims 7 to 9 
wherein the painting program further assigns to a 
customer, represented by the person's face, a 
number and assigns to a colour cosmetic a barcode 
symbol or other designation correlating an ordered 
colour cosmetic with the customer. 

11. An apparatus according to any of claims 7 to 10 
wherein the painting program further includes in- 
quiring with a customer represented by the person's 
face on their own colour preferences which input is 
then registered to change the colour of the located 
areas of the displayed facial image. 



PatentansprUche 

1. Verfahren zum Anwenden eines virtuellen Makeo- 
vers auf das Geslcht elner Person, wobel das Ver- 
fahren umfasst: 

(i) Richten einer Digitalkamera auf das Gesicht 
elner Person zur Aufnahme eines Bildes mln- 
destens eines Teils davon; 



(ii) Berechnen von Farbparametern auf zwei 
odermehreren Bereichen des Bildes, urn eine 
naturliche Farbung der Haut zu identifizieren; 

(iii) Ubertragen des Bildes auf einen Monitor zur 
5 Anzeige des Gesichts; 

(iv) Korrelieren der naturiichen Hautfarbe mit 
elner vorbestimmten Farbpalette, die zur Be- 
rechnung der naturiichen Hautfarbe geeignet 
ist; 

10 (v) Ausfindigmachen von Bereichen des Ge- 

sichts zur Anwendung eines farbigen Kosmeti- 
kums; und 

(vi) Abbilden des Gesichtsbildes mit der vorbe- 
stimmten Farbpalette auf den ausflndlg ge- 
15 machten Bereichen. 

2. Verfahren nach Anspruch 1, das auBerdem eine 
Bestellung farbiger Kosmetika entsprechend der 
vorbestimmten Farbpalette umfasst. 



20 

3. Verfahren nach Anspruch 1 Oder Anspruch 2, wobei 
die ausfindig gemachten Bereiche Lippen sind und 
in Schritt (vi) Anzeigen einer Lippen stiftmarkierung 
Oder einer Farbe, die sich von der naturiichen Haut- 

25 farbe unterscheidet, auf den Lippen. 

4. Verfahren nach Anspruch 1 oder Anspruch 2, wobei 
die ausfindig gemachten Bereiche sich urn die Au- 
gen befinden und in Schritt (vi) Anzeigen einer Lid- 
so strich- oder Lidschattenmarkierung in einer Faroe, 

die sich von der naturiichen Hautfarbe unterschei- 
det, urn die Augen. 

5. Verfahren nach einem der Anspruche 1 bis 4, das 
35 auBerdem den Schritt der Zuordnung einer Zahl ei- 
ner durch das Gesicht der Person rep rase ntierten 
Kundin und Zuordnen eines Strichcodesymbols 
oder einer anderen Bezeichnung, die bestellte far- 
bige Kosmetika mit der Kundin korreliert, den farbi- 

40 gen Kosmetika. 

6. Verfahren nach einem der Anspruche 1 bis 5, das 
auBerdem Befragen der durch das Gesicht der Per- 
son reprasentierten Kundin uber ihre eigenen Farb- 

45 bevorzugungen umfasst, wobei die Eingabe dann 
aufgenommen wird, urn die Farbe des ausfindig ge- 
machten Bereiches des angezeigten Gesichtsbil- 
des zu andem. 

so 7. Vorrichtung zum Anwenden eines virtuellen Makeo- 
vers auf das Gesicht einer Person, wobei die Vor- 
richtung umfasst: 

(i) eine Digitalkamera zum Aufnehmen von In- 
55 formationen hinstchtlich des Gesichtsbildes der 

Person; 

(II) ein Programm zur Berechnung der Farbe 
auf zwei oder mehreren verschiedenen Berei- 
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chen des Gesichtsbilds, die normalerweise 
nicht von Kosmetika bedeckt sind, urn die na- 
tiiriiche Farbe der Haut der Person zu identifi- 
zieren; 

(Hi) einen Monitor zur Anzeige des Gesichtsbil- 
des; 

(Iv) ein Programm zum Farbauftrag einer vor- 
bestimmten Farbpalette, die mit der naturlichen 
Hautfarbe auf bestimmten ausfindig gemach- 
ten Bereichen des Gesichts, die normalerweise 
Gegenstand elner farbkosmetischen Behand- 
lung sind, korrelieren; und 
(v) ein Drucker, der einen Ausdruck der vorbe- 
stimmten Farbpalette ausgibt. 

B. Vorrichtung nach Anspruch 1, wobei die ausfindig 
gemachten Bereiche Lippen sind und das Farbau- 
ftragprogramm auf den Lippen eine Lippenstiftmar- 
kierung mit einer Farbe anzeigt, die sich von jener 
der naturlichen Hautfarbe unterscheidet. 

9. Vorrichtung nach Anspruch 7. wobei die ausfindig 
gemachten Bereiche sich urn die Augen herum be- 
finden und das Farbauftragprogramm urn die Au- 
gen herum eine Lidstrich- oder eine Lldschatten- 
markierung einer Farbe, die sich von der naturli- 
chen Hautfarbe unterscheidet, anzeigt. 

10. Vorrichtung nach einem der Anspriiche 7 bis 9, wo- 
bei das Farbauftragprogramm auGerdem der durch 
das Gesicht der Person reprasentierten Kundin ei- 
ne Zahl zuordnet und ein Strlchcodesymbol oder ei- 
ne andere Bezeichnung, die ein angefordertes 
Farbkosmetikum mit der Kundin korreliert, einem 
farbigen Kosmetikum zuordnet. 

11. Vorrichtung nach einem der Anspriiche 7 bis 10, 
wobei das Farbauftragprogramm auBerdem Befra- 
gen elner durch das Gesicht der Person reprasen- 
tierten Kundin hinsichtlich ihrer eigenen Farbbevor- 
zugungen einschlieBt, wobei die Eingabe dann auf- 
genommen wird, urn die Farbe der ausfindig ge- 
machten Bereiche des angezelgten Gesichtsbildes 
zu andern. 



Revendicatlons 

1 . ProcedS ^application d'un changement de style vir- 
tue! sur le visage d'une perconne, ledit proc^decon- 
sistant a : 

(I) diriger une camera numerique vers le visage 
d'une person ne afin d'enregistrer une image 
d'au moins une partie de celui-ci ; 
(ii) calculer les parametres de couleur de deux 
zones ou plus de I'image afin d'identlfier une 
couleur naturelle de la peau ; 



(iii) transmettre I'image sur un ecran afin cfaffi- 
cher le visage ; 

(iv) correler la couleur naturelle de la peau avec 
une palette predetermine de couleurs appro- 

5 priees a la couleur naturelle calculee de la 

peau ; 

(v) localiser les zones du visage pour I'applica- 
tion d'un cosmetique colore ; et 

(vi) afficher I'image du visage avec la palette de 
10 couleur prSdeterminee su r les zones localisees 

2. ProcedS selon la revendication 1 , comprenant en 
outre le fait de passer une commande de cosmeti- 
ques colores correspondant a la palette de couleur 

15 predetermined. 

3. Procede selon la revendication 1 ou la revendica- 
tion 2, dans lequel les zones localisees sont les le- 
vres et, dans le cadre de I'etape (vi), afficher sur les 

20 levres une marque de rouge a levres ou une couleur 
autre que celle de la couleur naturelle de la peau. 

4. ProcSdS selon la revendication 1 ou la revendica- 
tion 2, dans lequel les zones localisees sont situSes 

25 autour des yeux et, dans le cadre de I'etape (vl), 
afficher autour des yeux une marque d'eyeliner ou 
d'ombre a paupieres d'une couleur diffSrente de la 
couleur naturelle de la peau. 

30 5. ProcSdeselonl'unequelconquedesrevendications 
1 a 4, comprenant en outre I'etape consistant a at- 
tribuer a un client, represents par le visage de la 
personne, un numero, et a attribuerau cosmetique 
colors un symbole de code barre ou une autre d6- 

35 signation qui fasse la correlation entre la comman- 
de de cosmetique colore et le client. 

6. Proced6 selon Tune quelconquedes revendications 
1 a 5, comprenant en outre le fait de demander a 

*o un client represents par le visage d'une personne 
quelles sont ses preferences personnelles en ma- 
ture de couleur, lesdites informations Stant ensuite 
enregtstrSes afin de changer la couleur de la zone 
localisSe de I'image du visage exposee. 

45 

7. Appareil pour appliquer un changement de style vlr- 
tuel sur le visage d'une personne, ledit appareil 
comprenant: 

so (j) une camera numSrique pour acquSrlr de I'in- 

formation sur I'image du visage d'une 
personne : 

(li) un programme pour calculer la couleur sur 
deux zones differentes ou plus de I'image du 
55 visage qui n'est pas nomnalement recouverte 

par des cosmStiques afin d'identifier une cou- 
leur naturelle d'un visage d'une personne ; 
(iii) un ecran pour afficher I'image du visage ; 



5 



9 



EP1 194 898 B1 



(iv) un programme pour peindre une palette de 
couleurs predeterminees en correlation avec la 
couleur natureile de la peau sur certaines zo- 
nes localisees du visage qui sont normalement 
soumises a un traitement cosmetique colore ; 5 
et 

(v) une Imprimante dellvrant une epreuve de la 
palette de couleur predeterminee. 

8. Apparel! selon la revendication 7 dans lequel les zo- io 
nes localisees sont les levres et le programme de 
peinture affiche sur les levres une marque de rouge 

a levres ayant une couleur differente de celle de la 
couleur natureile de la peau. 

15 

9. Appareil selon la revendication 7, dans lequel les 
zones localisees sont autour des yeux et le pro- 
gramme de peinture affiche autour des yeux une 
marque d'eyeliner ou d'ombre a paupieres d'une 
couleur autre que celle de la couleur natureile de la 20 
peau. 

10. Appareil selon Tune quelconque des revendications 
7 a 9, dans lequel le programme de peinture attribue 

en outre a un client, represents par le visage de la 25 
personne, un numero, et assigne un cosmetiqueco- 
lore a un symbole de code barre ou une autre de- 
signation qui fasse la correlation entre un cosmeti- 
que colore commande et un client. 

30 

1 1 . Appareil selon Tune quelconque des revendications 
7 a 10, dans lequel le programme de peinture Inclut 
en outre le fait de demander a un client represents 
par le visage d'une personne, quelles sont ses pre- 
ferences personnelles en matiere de couleur, ces 35 
informations etant ensuite enregistrees pour chan- 
ger la couleur des zones localisees de I'image de 
visage exposee. 
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